Stabilize Histogram to Prometheus translation and specify NHCB conversion#5091
Conversation
…fy NHCB conversion Mark the Histograms section of the Prometheus/OpenMetrics compatibility spec as Stable, and add a detailed specification for converting an OpenTelemetry Histogram to a Prometheus Native Histogram with Custom Buckets (NHCB) as an opt-in alternative to the classic Prometheus Histogram representation. Fixes open-telemetry#4605 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Clarify that the opt-in OpenTelemetry Histogram to NHCB conversion only applies when the target is Prometheus Remote-Write 2.0 or later, since OpenMetrics native histograms only support the exponential variant. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
ArthurSens
left a comment
There was a problem hiding this comment.
Krajo and I had a quick sync on this PR today, and Krajo shared some important nuances.
NHCB is much more efficient than Prometheus Classic histograms, so we should encourage using that format. However, it is only supported by Remote Write v2 (push-based) and is therefore irrelevant to the OTel SDKs, which implement the Prometheus pull-based exporter.
I believe Prometheus Remote Write exporter SHOULD default to converting to NHCB, but all others MUST convert to classic. @open-telemetry/prometheus-interoperability, any opinions here? We're leaking protocols into the compatibility spec, but I think it's fine 😬
|
Also fixes #4921 |
|
I'm tempted to try and remove the concept of NHCB from the PRW 2.0 protocol: prometheus/prometheus#18689 to make it internal to the Prometheus server. I think for now, we can add the NHCB spec, but leave it in development -- even if we otherwise stabilize native histograms or classic histogram sections. |
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Restructure the Histograms section so the classic Prometheus Histogram conversion remains Stable while the new NHCB conversion is introduced as a Development subsection. Move the Delta temporality rule to the Stable parent so it covers both paths, generalize the opt-in wording in the Stable parent, and note the current NHCB protocol constraint inside the Development subsection. Regenerate the TOC and split the CHANGELOG entry to reflect both the stabilization and the addition. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
|
I've reorganized the text to be able to stabilize the Prometheus (classic) Histograms existing bit and add NHCB as a "Development" stability bit. @dashpole @ArthurSens @jack-berg |
# Conflicts: # CHANGELOG.md
Bucket boundaries have the name "ExplicitBounds" in SDK/OTLP. Have max 80 character lines. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
conversion. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
|
I see this as two changes. The stabilize histogram part is good news, thanks. |
…stogram translation Mark the OTLP Exponential Histogram to Prometheus Native Histogram (standard exponential schema) conversion section as Stable. Restructure the bullets to mirror the Histograms section established in open-telemetry#5091, splitting the NoRecordedValue handling into explicit true/false branches. Fix the NoRecordedValue handling: the Native Histogram Count is a uint64 and MUST be set to zero (not the Stale NaN value) on staleness, with ZeroCount/PositiveSpans/PositiveDeltas/NegativeSpans/NegativeDeltas left empty. Set the Native Histogram ResetHint to UNKNOWN since OpenTelemetry does not carry an explicit reset flag. Add StartTimeUnixNano handling consistent with the Histograms section, and note that fields not explicitly referenced are left at their zero value. Relax the delta-temporality rule from "MUST be dropped" to "MAY be aggregated to cumulative or MUST be dropped", matching the (now Stable) Histograms section. Fixes open-telemetry#4922 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Mirror the detailed explanation added for the OpenTelemetry Histogram to Prometheus NHCB conversion in open-telemetry#5091: split the sparse-layout conversion into separate bullets covering which counts are converted, delta encoding, span Offset/Length semantics, and a worked example. Provide an explicit formula (`Offset + i + 1`) for mapping an Exponential Histogram bucket array position to a Native Histogram bucket index, since the indexing convention differs by one between the two formats. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Mirror the detailed explanation added for the OpenTelemetry Histogram to Prometheus NHCB conversion in open-telemetry#5091: split the sparse-layout conversion into separate bullets covering which counts are converted, delta encoding, span Offset/Length semantics, and a worked example. Provide an explicit formula (`Offset + i + 1`) for mapping an Exponential Histogram bucket array position to a Native Histogram bucket index, since the indexing convention differs by one between the two formats. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
There was a problem hiding this comment.
LGTM, but with one nit.
The Histograms section is marked as stable, even though one of its subsections is not yet stable. I would just change this:
### Histograms
// stable <------ should be development
#### Histograms as Prometheus histograms
// stable
#### Histograms as Prometheus NHCB
// development
there's also a question about reset hints from josh that deserves an answer :)
Co-authored-by: David Ashpole <dashpole@google.com>
|
I believe this is ready to merge. |
|
Markdown failure is due to github.com returning 504 gateway timeout |
|
@dashpole do you have any remaining thoughts? Will leave it to you to approve / merge. |
efceef7
### Context - Clarify that environment variable propagation carriers normalize requested keys, carrier keys, and returned keys. ([open-telemetry#5102](open-telemetry#5102)) - Specify that environment variable propagation carriers only read and return normalized environment variable names. ([open-telemetry#5144](open-telemetry#5144)) - Specify that an empty environment variable propagation name is non-normalized and normalizes to `_`. ([open-telemetry#5163](open-telemetry#5163)) ### Profiles - Remove duplicate information from and extend Profiles documentation (README.md, pprof.md). ([open-telemetry#4932](open-telemetry#4932)) ### Entities - Add specification for communicating entity information as structured log events. ([open-telemetry#4836](open-telemetry#4836)) ### Common - Add an in-development [SDK self-observability](specification/self-observability.md) section, referenced from the Tracing, Metrics, and Logs SDK specs. ([open-telemetry#5087](open-telemetry#5087)) - Clarify non-OTLP representation guidance for nested `AnyValue` values in arrays and maps. ([open-telemetry#5053](open-telemetry#5053)) - Add in-development guidance recommending a JSON object as the string representation for an attribute in non-OTLP protocols. ([open-telemetry#5028](open-telemetry#5028)) - Add in-development guidance recommending a JSON object as the string representation for an attribute collection in non-OTLP protocols. ([open-telemetry#5110](open-telemetry#5110)) ### Compatibility - Deprecate OpenCensus compatibility requirements in the specification. ([open-telemetry#5138](open-telemetry#5138)) - Stabilize sections of Prometheus Metrics Exporter. - Clarify resource attributes configuration. ([open-telemetry#5084](open-telemetry#5084)) - Stabilize the conversion of OTLP Summaries into Prometheus Summaries. ([open-telemetry#5107](open-telemetry#5107)) - Stabilize client libs section. ([open-telemetry#5106](open-telemetry#5106)) - Stabilize Prometheus Metrics Exporter default aggregation configuration. ([open-telemetry#5113](open-telemetry#5113)) - Stabilize sections of Prometheus and OpenMetrics Compatibility. - Stabilize OpenTelemetry Histogram to Prometheus Histogram transformation. ([open-telemetry#5091](open-telemetry#5091)) - Add optional OpenTelemetry Histogram to Prometheus Native Histogram with Custom Buckets transformation. ([open-telemetry#5091](open-telemetry#5091)) ### SDK Configuration - Add link to declarative config IdGenerator type ([open-telemetry#5133](open-telemetry#5133)) ### OTEPs - Context-scoped Attributes. ([open-telemetry#4931](open-telemetry#4931)) --------- Co-authored-by: Reiley Yang <reyang@microsoft.com>
Fixes #4605
Changes
Mark the Histograms section of the Prometheus and OpenMetrics compatibility specification as Stable, and add a detailed specification for converting an OpenTelemetry Histogram to a Prometheus Native Histogram with Custom Buckets (NHCB) as an opt-in alternative to the classic Prometheus Histogram representation. The NHCB output is scoped to the Prometheus Remote-Write 2.0 (or later) protocol, since OpenMetrics native histograms only support the exponential variant.
The classic-Histogram conversion bullets are reorganized for symmetry with the new NHCB section. The NHCB section covers
Schema,Count/Sum(includingNoRecordedValuehandling),Timestamp,StartTime,CustomValues(with+Infoverflow-bucket handling),PositiveSpans/PositiveDeltas(sparse encoding, delta semantics, negative-boundary handling),ResetHint, flavor, exemplars, and which other fields must be left at their zero values.Notes for reviewers:
Countis set to Stale NaN onNoRecordedValue. For NHCB integer-counter flavor,Countis auint64so this PR uses0instead. The ExponentialHistogram wording likely deserves the same treatment in a follow-up PR.CHANGELOG.mdfile updated for non-trivial changes[chore]in the PR title to skip the changelog checkCoded with Claude Opus 4.7.